Unified workstation for virtual craniofacial diagnosis, treatment planning and therapeutics

ABSTRACT

An integrated system is described in which digital image data of a patient, obtained from a variety of image sources, including CT scanner, X-Ray, 2D or 3D scanners and color photographs, are combined into a common coordinate system to create a virtual three-dimensional patient model. Software tools are provided for manipulating the virtual patient model to simulation changes in position or orientation of craniofacial structures (e.g., jaw or teeth) and simulate their affect on the appearance of the patient. The simulation (which may be pure simulations or may be so-called “morphing” type simulations) enables a comprehensive approach to planning treatment for the patient. In one embodiment, the treatment may encompass orthodontic treatment. Similarly, surgical treatment plans can be created. Data is extracted from the virtual patient model or simulations thereof for purposes of manufacture of customized therapeutic devices for any component of the craniofacial structures, e.g., orthodontic appliances.

RELATED APPLICATIONS

This application claims priority benefits pursuant to 35 U.S.C. §120 asa continuation of application Ser. No. 10/429,123 filed May 02, 2003 nowU.S Pat. No. 7,234,937, which is a continuation-in-part of applicationSer. No. 10/340,404 filed Jan. 9, 2003, now abandoned, which is acontinuation of application Ser. No. 09/560,641, filed Apr. 28, 2000,now U.S. Pat. No.6,512,994, which is a continuation-in-part ofapplication Ser. No. 09/452,034 filed Nov. 30, 1999, abandoned. Thisapplication also claims priority benefits pursuant to 35 U.S.C. §120 asa continuation-in-part of application Ser. No. 09/835,039 filed Apr. 13,2001, now U.S. Pat. No. 6,648,640. The entire contents of the relatedapplications are fully incorporated by reference herein.

This application is also related to a patent application filed May 02,2003, inventors Rohit Sachdeva et al., entitled INTERACTIVE UNIFIEDWORKSTATION FOR BENCHMARKING AND CARE PLANNING, Ser. No. 10/429,074,pending, the entire contents of which are incorporated by referenceherein.

This application is also related to a patent application filed May 02,2003, inventors Rohit Sachdeva et al., entitled METHOD AND SYSTEM FORINTEGRATED ORTHODONTIC TREATMENT PLANNING USING UNIFIED WORKSTATION,Ser. No. 10/428,461, pending, the entire contents of which areincorporated by reference herein.

BACKGROUND OF THE INVENTION

A. Field of the Invention

This invention relates to the field of computerized techniques fordiagnosis and planning medical and dental treatment of human patients.More particularly, the invention is directed to a unified workstationand associated computerized techniques for creating a virtualthree-dimensional model of the patient, including bone, soft tissue, andteeth from data from a variety of diverse imaging sources. The inventionis also related to computer software tools enabling a user to use such avirtual model for diagnosis and planning treatment of craniofacialstructures of the patient, including teeth, and for export of data todiverse manufacturers of therapeutic devices for the patient, such asorthodontic appliances.

B. Description of Related Art

The diagnosis and treatment of patients with craniofacial problems ordisease typically begins with the obtaining of clinical history, medicalhistory, dental history, ultrasonic scanned images, 2D or 3D scannedimages, photographs, and 2D or 3D X-rays. Such X-rays are taken from thefront and the side view. X-rays are also taken to show the condition ofthe teeth and the jaws. At this stage, diagnosis and treatment planningis often done by the practitioner on a sheet of acetate over the X-rays.Generally, this process is not very scientific, and it is time consumingand requires experience. There is no guarantee how good the results ofthe treatment will be. Similarly, orthodontists typically mentallyvisualize a target or desired occlusion for an orthodontic patient andattempt to bend archwires by hand to move teeth to the desired position.This approach also lacks reliability, reproducibility and precision.

More sophisticated, computer-based approaches to diagnosis and treatmentplanning of craniofacial structures, including the straightening ofteeth, have been proposed. See Andreiko, U.S. Pat. No. 6,015,289; Snow,U.S. Pat. No. 6,068,482; Kopelmann et al., U.S. Pat. No. 6,099,314;Doyle, et al., U.S. Pat. No. 5,879,158; Wu et al., U.S. Pat. No.5,338,198, and Chisti et al., U.S. Pat. Nos. 5,975,893 and 6,227,850,the contents of each of which is incorporated by reference herein. Also,imaging and medical diagnostic software and related products aremarketed by Dolphin Imaging, 661 Independence Avenue, Canoga Park,Calif. 91309-2944. A method for generation of a 3D model of thedentition from an in-vivo scan of the patient, and interactivecomputer-based treatment planning for orthodontic patients, is describedin published PCT patent application of OraMetrix, Inc., the assignee ofthis invention, publication no. WO 01/80761, the contents of which areincorporated by reference herein. Other background references related tocapturing three dimensional models of dentition and associatedcraniofacial structures include S. M. Yamany and A. A. Farag, “A Systemfor Human Jaw Modeling Using Intra-Oral Images” in Proc. IEEE Eng. Med.Biol. Soc. (EMBS) Conf, Vol. 20, Hong Kong, October 1998, pp. 563-566;and M. Yamany, A. A. Farag, David Tasman, A. G. Farman, “A 3-DReconstruction System for the Human Jaw Using a Sequence of OpticalImages,” IEEE Transactions on Medical Imaging, Vol. 19, No. 5, May 2000,pp. 538-547. The contents of these references are incorporated byreference herein.

The technical literature further includes a body of literaturedescribing the creation of 3D models of faces from photographs, andcomputerized facial animation and morphable modeling of faces. See,e.g., Pighin et al., Synthesizing Realistic Facial Expression fromPhotographs, Computer Graphics Proceedings SIGGRAPH '98, pp. 78-94(1998); Pighin et al., Realistic Facial Animation Using Image-based 3DMorphing, Technical Report no. UW-CSE-97-01-03, University of Washington(May 9, 1997); and Blantz et al., A Morphable Model for The Synthesis of3D Faces, Computer Graphics Proceedings SIGGRAPH '99 (August, 1999). Thecontents of these references are incorporated by reference herein.

The art has lacked a truly integrated and unified system in which softtissue (skin, lips, etc.) and the underlying bone and other craniofacialfeatures, including teeth, are superimposed and registered together in acommon coordinate system to create a complete virtual patient model thatalso includes the exterior appearance of the patient, and in which theuser is provided with tools to study the interaction of suchcraniofacial features to each other and to simulate with a computerchanges in craniofacial features (such as by means of proposed toothextraction, orthodontic manipulation, or surgery) and their effects onthe external, visual appearance of the patient, and design optimaltherapeutics based upon the unified virtual patient.

A principal benefit of the invention is that it provides a powerful toolto the physician, dentist or orthodontist for diagnosis and treatmentplanning. The unified workstation provides comprehensive, multiplefunctionalities in the same unit, thus eliminating the need for moreexpensive and less efficient multiple workstations wherein eachworkstation is dedicated to performing one specific function or alimited sub-set of functions necessary for the practitioner's practice.Moreover, the three-dimensional virtual patient model described hereinis useful datum for use in a diverse set of possible treatment regimesfor treatment of the patient. As such, the virtual patient model (orperhaps some subset of data from the model) can be provided or exportedto manufacturers of appliance systems for their use in designing and/orfabricating customized appliances for treatment of the patient, e.g.,customized orthodontic appliances.

SUMMARY OF THE INVENTION

In a first aspect, a system for use in diagnosis and planning treatmentof a human patient is provided. The system includes a general-purposecomputer system having a processor (e.g., central processing unit) and auser interface. The details of the computer system are not important. Amemory is provided which is accessible to the general-purpose computersystem, such as a hard disk or a file server on a network to which thegeneral-purpose computer is connected. The memory stores a first set ofdigital data representing patient craniofacial image informationobtained from a first imaging device. For example, the first set ofdigital data may be 3-D scan data obtained from a scan of the patient'sface using a scanner, 3D scan data from a scan of the dentition of thepatient, X-ray data, CT scan, MRI, video, a set of two-dimensionaldigital color photographs of the patient, etc. The memory furtherincludes a second set of digital data representing patient craniofacialimage information obtained from a second image device different from thefirst image device. For example, if the first set of data represents CTscan data, the second set of data may represent 3D scan data of theteeth of the patient. The first and second sets of data represent, atleast in part, common craniofacial anatomical structures of the patient.In other words, there are some anatomical features that are common tothe two sets of data; they overlap to some extent. One of the first andsecond sets of data will typically include data representing the surfaceconfiguration or external appearance of the patient's face, for examplea two dimensional digital photograph of the face (black and white orcolor), a 3D scan of the face, or other face data.

The system further includes a set of computer instructions stored on amachine-readable storage medium accessible to said general-purposecomputer system. The computer instructions need not necessarily bestored on the same memory as the first and second sets of data. In theillustrated embodiment, the instructions are stored in the hard diskmemory of the general-purpose computer system and are executed by thecomputer's host processor, but that need not always be the case. The setof instructions cause the general purpose computer system to performseveral tasks:

-   -   1) Firstly, automatically, and/or with the aid of operator        interaction, the set of instructions includes instruction that        operate to superimpose the first set of digital data and the        second set of digital data so as to provide a composite,        combined digital representation of the craniofacial anatomical        structures in a common. Preferably, but not necessarily, this        representation will be a three-dimensional representation in a        common 3D coordinate system. This representation is referred to        herein occasionally as a “virtual patient model.” In this        aspect, the techniques of creation of a 3-D model disclosed in        the patent application of Rohit Sachdeva et al., Ser. No.        09/560,641 filed Apr. 28, 2000 may be employed. Scaling        techniques may be used to scale the data from one set of images        to the other so as to created correctly scaled composite model        that accurately reflects the patient's anatomy.    -   2) Secondly, the instructions include instructions for        displaying the composite, combined digital representation of the        craniofacial anatomical structures to a user of the system, for        example on the user interface of the general purpose computer        system.        Preferably, the instructions include instructions providing the        user with tools on the user interface for visually studying the        interaction of the craniofacial anatomical structures and their        relationship to the external, visual appearance of the patient.        The set of tools including tools for simulating changes in the        anatomical position or shape of the craniofacial anatomical        structures and measuring their effect on the external, visual        appearance of the patient.

In a representative embodiment, 3D data of the face, skull and jaw isobtained from various scanning or imaging devices (CT scan, X-Ray, colorphotographs) and stored in the memory. Then, the general-purposecomputer superimposes the data to place all the data in one commoncoordinate system to create a virtual patient model. Scaling of the datamay be performed in this step. The virtual patient model is displayed tothe user of the system. The software instructions in the system providemodeling or “morphing” tools which allow the user to manipulate variousparameters and simulate the effect of such changes on the appearance ofthe patient, such as the position of one or more teeth or jaw, the shapeof the arches, the age of the patient; the color and texture of theteeth; and the reflectivity and ambient conditions of the light shiningon the patient.

In another aspect of this invention, an orthodontic treatment planningsystem is provided comprising a 3D scanner for scanning the dentition ofthe patient, a general-purpose computer receiving scan data from thescanner and responsively generating a three-dimensional virtual model ofthe dentition of the patient, and software stored on a machine-readablememory accessible to the general-purpose computer. The software containsinstructions for combining, either automatically or with the aid of anoperator, scan data from the scanner with digital data of the facialappearance of the patient. The digital data of the facial appearance ofthe patient can be obtained from a variety of sources, such as colordigital camera or from a scanning of the face with the 3D scanner. Thesoftware combines (e.g., superimposes) the two sets of data to therebycreate a combined digital three-dimensional representation of thedentition and the facial appearance in a common three-dimensionalcoordinate system.

The software further includes instructions providing the user with toolsto manipulate the position of the virtual teeth in the three-dimensionalvirtual model of the dentition relative to other anatomical structuresof the patient and to visualize the effect of proposed changes in toothposition on the facial appearance of the patient. Thus, the toolsthereby provide the user with the ability to design with the computer adesired three-dimensional configuration of the virtual teeth whileviewing the effect of changing tooth position on the visual appearanceof the face of the patient.

In a preferred embodiment, the scanner comprises a hand-held,three-dimensional optical scanner. The digital data of the facialappearance of the patient can be obtained from the hand-held,three-dimensional optical scanner, thereby obviating the need for anyother data acquisition devices. On the other hand the digital data couldbe obtained from a color camera, a video camera, or other type ofimaging or scanning device. Other types of imaging devices could beused, such as radiographic images, CAT scan images, or MRI images.

In one possible embodiment, the system can include software combiningthe digital three-dimensional representation of the dentition and facialappearance with X-ray data superimposed on the scan data and the digitaldata of the facial appearance of the patient.

With the system of this invention, the elements of the craniofacialdental complex can be analyzed quickly in either a static or dynamicformat, using the unified workstation and simulation tools provided insoftware in the workstation. The virtual patient model enables thesimulation of facial expressions such as smiling, grimacing, the agingof the patient, and functional movements such as chewing and othercomplex motions of the jaw, in both a static manner and in a dynamicmanner. For example, the virtual patient model is displayed and currentsmile of the patient is viewed, and changes to the smile are simulated,as for example by the simulation of tooth movement and its effect onsoft tissue, lips etc. and its effect on the smile. The simulationscould be performed as a dynamic simulation, in which the series ofchanges in tooth position (intermediate positions), and their effect onsoft tissue during the smile, is demonstrated in a manner showing themotion of the teeth and tissues. Alternatively, the simulations could bestatic, for example, movement of one or more teeth from one position toanother is performed, and the virtual patient model is shown with theeffect on that movement on the change in soft tissue configuration(e.g., lip) or on the overall smile. There is also a possibility ofsimulations in between purely static simulations and dynamicsimulations, such as stepping through a series of intermediate positionsone at a time, essentially breaking the dynamic simulation down into aseries of steps.

In the above simulations, the teeth of the patient are preferablyrepresented as individual tooth models that are moveable relative toeach other. The clinician is provided with tools to manipulate theirposition for diagnostic and treatment-planning purposes. Moreover, thetools provide the user the ability to simulate changes in the positionor shape of the jaw, tooth or teeth, and the movement of such structuresand the skull movement, and to visually observe the effect of suchsimulated changes on the patient's face and smile. This provides forpowerful tools for study of proposed treatments for the patient.Similarly, the patient's desired feature and smile can be simulated onthe user interface, and from that desired feature and smile it ispossible to automatically back solve for the required jaw, and/or toothmovements or changes needed to provide that desired result, simply bycomparing “before” and “after” positions of the jaw, tooth and/or skullpositions.

Thus, in the broader aspects, we have invented an apparatus forassembling a virtual patient model from various data sources including3D scanners, X-rays and 2D color camera. We have also invented a uniquemethod for studying the interaction of craniofacial structures byvarying the smile and age of the patient and the position of the teeth.For example, for the desired smile on the face, the best position of theteeth is calculated. On the other hand, the effect of various positionsof the teeth on the smile can also be studied. Furthermore, we haveinvented a unique method for treatment planning of craniofacialstructures based on the virtual patient model. By modifying variousparameters, we can create multiple morphed models and multiple treatmentplans quickly and reliably.

In presently preferred embodiments, the workstation also providescapabilities for integrating two and three-dimensional image data from avariety of sources, accessing treatment planning tools (software) eitherdirectly in the workstation or by furnishing data to a separateworkstation that has such software, and integrating the resultingappliance design and treatment plan in a form compatible with thecomputer systems of diverse appliance manufacturers. In essence, theworkstation facilitates a common platform by which a practitioner canintegrate the acquisition of data, the treatment plan, and the appliancedesign and manufacture into one seamless system.

BRIEF DESCRIPTION OF THE DRAWINGS

Presently preferred embodiments of the invention are described below inreference to the appended drawings, wherein like reference numeralsrefer to like elements in the various views, and in which:

FIG. 1 is block diagram of a system for creating a three-dimensionalvirtual patient model and for diagnosis and planning treatment of thepatient.

FIG. 2 is a flow chart showing a method of three-dimensional facecreation from scanning systems, which may be executed in software in thecomputer system of FIG. 1.

FIG. 3 is a flow chart showing an alternative method ofthree-dimensional face model face creation using a plurality of possibleinput image or data formats, which may be executed in software in thecomputer system of FIG. 1.

FIG. 4 is a flow chart showing a method of creating a complete texturedthree-dimensional model of teeth; simulations, either static or dynamic,can be performed with the textured 3D tooth models to simulate proposedtreatments and their effect on the patient's visual appearance.

FIGS. 4A-4E show a technique for combining 2D color photographs with 3Dtooth data to created textured (colored) 3D tooth models.

FIG. 5 is a screen shot of the user interface of FIG. 1 showing athree-dimensional face model and a three-dimensional tooth model, inseparate coordinate systems (i.e., prior to registration orsuperposition of the two relative to each other). FIG. 5 also shows aplurality of icons, which, when activated, provide tools formanipulating the models shown in the Figure.

FIG. 6 is a screen shot showing one possible method of placement of thelower jaw 3D data into the face data coordinate system usingcorresponding points that are common to each data set.

FIG. 7 is a screen shot showing the face data and the lower jaw 3D datain a common coordinate system (the face coordinate system of FIGS. 5 and6).

FIG. 8 is a screen shot showing the face data and skull data obtainedfrom a CT scan in a common coordinate system.

FIG. 9 is a screen shot showing face data and skull data superimposed onX-ray data obtained from the patient.

FIG. 10 is a screen shot showing the superposition of skull and facedata with X-Ray data.

FIGS. 11A-11E are a series of views of a digital model of an orthodonticpatient obtained, for example from CT scan, photographs, or intra-oralscanning with a hand-held 3D scanner.

FIG. 12 is a diagram illustrating a technique for scaling orthodonticdata obtained from an imaging device, such as a camera, to the actualanatomy of the patient.

FIG. 13 is a diagram showing an alternative scaling method similar tothat shown in FIG. 12.

FIG. 14 is an illustration of an X-ray of a set of teeth and adjacentbone.

FIG. 15 is an illustration of scaling the X-ray data of the tooth to theactual size of the tooth to produce a scaled digital model of the tooth.

FIGS. 16A-16C is an illustration of a method of determining orientationreference points in a digital model of a patient.

FIG. 17 is an illustration of a method of mapping the orientationreference points of FIGS. 16A-16C to a three-dimensional coordinatesystem.

FIG. 18 is an illustration of a method of mapping the orientationreference points of FIGS. 16A-16C to a three-dimensional coordinatesystem.

FIG. 19 is a more detailed block diagram of treatment planning softwareexecuted by the workstation of FIG. 1.

FIG. 20 is an illustration of the integration of the patient dataacquisition, treatment planning and appliance design functions that arefacilitated by a preferred embodiment of the unified workstation.

FIG. 21 is an illustration of screen display from the unifiedworkstation showing a 3D model of teeth in a proposed tooth position forthe patient; the 3D model that is flattened into a two-dimensionalrepresentation and placed in approximate registration with atwo-dimensional panorama X-ray photograph, thereby assisting the user inobtaining a better understanding between the tooth roots in the currentsituation and the proposed tooth position in a final or ideal situation.

FIG. 22 is a view of the screen display similar to FIG. 21, but with theX-ray hidden so as to only show the teeth in the proposed position.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

General Description

A unified workstation environment and computer system for diagnosis,treatment planning and delivery of therapeutics, especially adapted fortreatment of craniofacial structures, is described below. In onepossible example, the system is particularly useful in diagnosis andplanning treatment of an orthodontic patient. Persons skilled in the artwill understand that the invention, in its broader aspects, isapplicable to other craniofacial disorders or conditions.

A presently preferred embodiment is depicted in FIG. 1. The overallsystem 100 includes a general-purpose computer system 10 having aprocessor (CPU 12) and a user interface 14, including screen display 16,mouse 18 and keyboard 20. The system is useful for planning orthodontictreatment for a patient 34. In another example, the system isparticularly useful in planning therapeutics and designing customizedappliances for the patient. In still another example, the system isparticularly useful in integrating the required set of patientinformation (image data, clinical history data, etc.) to form a basisfor treatment planning. The virtual patient can be used in all facets ofdental care, such as planning surgical treatment, restorative dentistry,prosthodontics, design and manufacture of implants, etc.

The system 100 includes a memory 22 accessible to the general-purposecomputer system 10. The memory 22 stores two or more sets of digitaldata representing patient craniofacial image information. These setsinclude at least a first set of digital data 24 representing patientcraniofacial image information obtained from a first imaging device anda second set of digital data 26 representing patient craniofacial imageinformation obtained from a second image device different from the firstimage device. The first and second sets of data represent, at least inpart, common craniofacial anatomical structures of the patient. At leastone of the first and second sets of digital data normally would includedata representing the external visual appearance or surfaceconfiguration of the face of the patient.

In a representative and non-limiting example of the data sets, the firstdata set 24 could be a set of two dimensional color photographs of theface and head of the patient obtained via a color digital camera 28, andthe second data set is three-dimensional image information of thepatient's teeth, acquired via a suitable scanner 30, such as a hand-heldoptical 3D scanner, or other type of scanner. The memory 22 may alsostore other sets 27 of digital image data, including digitized X-rayphotographs, MRI or ultrasound images, CT scanner etc., from otherimaging devices 36. The other imaging devices need not be located at thephysical location per se of the workstation system 100. Rather, theimaging of the patient 34 with one or other imaging devices 36 could beperformed in a remotely located site (for example, at a clinic orhospital), in which case the image data is obtained by the workstation100 over the Internet 37 or some other communications medium, and storedin the memory 22.

The system 100 further includes a set of computer instructions stored ona machine-readable storage medium. The instructions may be stored in thememory 22 accessible to the general-purpose computer system 10. Themachine-readable medium storing the instructions may alternatively be ahard disk memory 32 for the computer system 10, external memory devices,or may be resident on a file server on a network connected to thecomputer system, the details of which are not important. The set ofinstructions, described in more detail below, comprise instructions forcausing the general computer system 10 to perform several functionsrelated to the generation and use of the virtual patient model indiagnostics, therapeutics and treatment planning.

These functions include a function of automatically, and/or with the aidof operator interaction via the user interface 14, superimposing thefirst set 24 of digital data and the second set 26 of digital data so asto provide a composite, combined digital three-dimensionalrepresentation of the craniofacial anatomical structures in a commonthree-dimensional coordinate system. This composite, combined digitalthree-dimensional representation is referred to herein occasionally asthe “virtual patient model,” shown on the display 16 of FIG. 1 as adigital model of the patient 34. Preferably, one of the sets 24, 26 ofdata includes photographic image data of the patient's face, teeth andhead, obtained with the color digital camera 28. The other set of datacould be intra-oral 3D scan data obtained from the hand-held scanner 30,CT scan data, X-Ray data, MRI, etc. The scan could be of a model of theteeth or of a facial moulage. In the example of FIG. 1, the hand-heldscanner 30 acquires a series of images containing 3D information andthis information is used to generate a 3D model in the scanning node 31,in accordance with the teachings of the published PCT application ofOraMetrix, PCT publication no. WO 01/80761, the content of which isincorporated by reference herein. Additional data sets are possible, andmay be preferred in most embodiments. For example the virtual patientmodel could be created by a superposition of the following data sets:intra-oral scan of the patient's teeth, gums, and associated tissues,X-Ray, CT scan, intra-oral color photographs of the teeth to add truecolor (texture) to the 3D teeth models, and color photographs of theface, that are combined in the computer to form a 3D morphable facemodel. These data sets are superimposed with each other, withappropriate scaling as necessary to place them in registry with eachother and at the same scale. The resulting representation can be storedas 3D point cloud representing not only the surface on the patient butalso interior structures, such as tooth roots, bone, and otherstructures. In one possible embodiment, the hand-held in-vivo scanningdevice is used which also incorporates a color CCD video camera tocapture either static, dynamic or video images, which may be eitherblack and white or in color.

The software instructions further includes a set of functions orroutines that cause the user interface 16 to display the composite,combined digital three-dimensional representation of craniofacialanatomical structures to a user of the system. In a representativeembodiment, computer-aided design (CAD)-type software tools are used todisplay the model to the user and provide the user with tools forviewing and studying the model. Preferably, the model is cable of beingviewed in any orientation. Tools are provided for showing slices orsections through the model at arbitrary, user defined planes.Alternatively, the composite digital representation may be printed outon a printer or otherwise provided to the user in a visual form.

The software instructions further include instructions that, whenexecuted, provide the user with tools on the user interface 14 forvisually studying, on the user interface, the interaction of thecraniofacial anatomical structures and their relationship to theexternal, visual appearance of the patient. For example, the toolsinclude tools for simulating changes in the anatomical position or shapeof the craniofacial anatomical structures, e.g., teeth, jaw, bone orsoft tissue structure, and their effect on the external, visualappearance of the patient. The preferred aspects of the software toolsinclude tools for manipulating various parameters such as the age of thepatient; the position, orientation, color and texture of the teeth;reflectivity and ambient conditions of light and its effect on visualappearance. The elements of the craniofacial and dental complex can beanalyzed quickly in either static format (i.e., no movement of theanatomical structures relative to each other) or in a dynamic format(i.e., during movement of anatomical structures relative to each other,such as chewing, occlusion, etc.). Intermediate levels of dynamicsimulation are possible, as explained previously.

The workstation environment provided by this invention provides apowerful system and for purposes of diagnosis, treatment planning anddelivery of therapeutics. For example, the effect of jaw and skullmovement on the patient's face and smile can be studied. Similarly, themodel can be manipulated to arrive at the patient's desired feature andsmile. From this model, and more particularly, from the location andposition of individual anatomical structures (e.g., individual toothpositions and orientation, shape of arch and position of upper and lowerarches relative to each other), it is possible to automatically backsolve for or derive the jaw, tooth, bone and/or soft tissue correctionsthat must be applied to the patient's initial, pre-treatment position toprovide the desired result. This leads directly to a patient treatmentplan.

These simulation tools, in a preferred embodiment, compriseuser-friendly and intuitive icons 35 that are activated by a mouse orkeyboard on the user interface of the computer system 10. When theseicons are activated, the software instruction provide pop-up, menu, orother types screens that enable a user to navigate through particulartasks to highlight and select individual anatomical features, changetheir positions relative to other structures, and simulate movement ofthe jaws (chewing or occlusion). Examples of the types of navigationaltools, icons and treatment planning tools for a computer user interfacethat may be useful in this process and provide a point of departure forfurther types of displays useful in this invention are described in thepatent application of Rudger Rubbert et al., Ser. No. 09/835,039 filedApr. 13, 2001, the contents of which are incorporated by referenceherein. Additional aspects of treatment planning that are possible areset forth in the patent application of Rohit Sachdeva et al., filed onthe same date as this application, entitled METHOD AND SYSTEM FORINTEGRATED ORTHODONTIC TREATMENT PLANNING USING UNIFIED WORKSTATION,Ser. No. 10/428,461, the content of which is incorporated by referenceherein.

The virtual patient model, or some portion thereof, such as datadescribing a three-dimensional model of the teeth in initial and targetor treatment positions, is useful information for generating customizedorthodontic appliances for treatment of the patient. The position of theteeth in the initial and desired positions can be used to generate a setof customized brackets, and customized archwire, which may be flatplanar or otherwise in shape, and customized bracket placement jigs, asdescribed in the above-referenced Andreiko et al. patents.Alternatively, the initial and final tooth positions can be used toderive data sets representing intermediate tooth positions, which areused to fabricate transparent aligning shells for moving teeth to thefinal position, as described in the above-referenced Chisti et al.patents. The data can also be used to place brackets and design acustomized archwire as described in the previously cited applicationSer. No. 09/835,039. Furthermore, surgical devices such as surgicalarchwires, splints, prosthetic devices, and restorative devices can befabricated with these data sets. Methods of fabricated customizedarchwires from data sets indicating bracket position and tooth geometryare disclosed in the patent application of Werner Butscher et al., Ser.No. 09/834,967. allowed, which is incorporated by reference herein.Methods of fabricating bracket placement jigs are described in. U.S.patent application Ser. No. 09/560,127, allowed, the contents of whichare incorporated by reference herein.

To facilitate sharing of the virtual patient model among specialists anddevice manufacturers, the system 100 includes software routines andappropriate hardware devices for transmitting the virtual patient modelor some subset thereof over a computer network. The system's softwareinstructions are preferably integrated with a patient management programhaving a scheduling feature for scheduling appointments for the patient.The patient management program provides a flexible scheduling of patientappointments based on progress of treatment of the craniofacialanatomical structures. The progress of treatment can be quantified. Theprogress of treatment can be monitored by periodically obtaining updatedthree-dimensional information regarding the progress of treatment of thecraniofacial features of the patient, such as by obtaining updated scansof the patient and comparison of the resulting 3D model with theoriginal 3D model of the patient prior to initiation of treatment.

Thus, it is contemplated that system described herein provides a set oftools and data acquisition and processing subsystems that togetherprovides a flexible, open platform or portal to a variety of possibletherapies and treatment modalities, depending on the preference of thepatient and the practitioner. For example, a practitioner viewing themodel and using the treatment planning tools may determine that apatient may benefit from a combination of customized orthodonticbrackets and wires and removable aligning devices. Data from the virtualpatient models is provided to diverse manufacturers for coordinatedpreparation of customized appliances. Moreover, the virtual patientmodel and powerful tools described herein provide a means by which thecomplete picture of the patient can be shared with other specialists(e.g., dentists, maxilla-facial or oral surgeons, cosmetic surgeons,other orthodontists) greatly enhancing the ability of diversespecialists to coordinate and apply a diverse range of treatments toachieve a desired outcome for the patient. In particular, the overlay orsuperposition of a variety of image information, including X-Ray, 3Dteeth image data, photographic data, CT scan data, and other data, andthe ability to toggle back and forth between these views and simulatechanges in position or shape of craniofacial structures, and the abilityto share this virtual patient model across existing computer networks toother specialists and device manufacturers, allows the entire treatmentof the patient to be simulated and modeled in a computer. Furthermore,the expected results can be displayed before hand to the patient andchanges made depending on the patient input.

With the above general description in mind, additional details ofpresently preferred components and aspects of the inventive system andthe software modules providing the functions referenced above will bedescribed next. The treatment plans developed using the virtual patientmodel and the unified workstation can be ones in which only one type ofappliance is used to treat the patient (such as brackets and wires) orhybrid treatment plans in which multiple types or classes of appliancesare used to treat the patient. Examples of hybrid treatment plansinclude plans in which both brackets and wires and removable aligningshells (see the Chisti et al. patents cited previously) are used duringthe course of treatment. The brackets and wires and removable appliancescould be used at the same type for different teeth, or they could beused at different times, or both could occur.

Capture of Image Information

The creation of the virtual patient model uses the capture and storageof at least two different digital sets of image data of the patient. Theimage sets will typically represent, at least in part, overlappingcraniofacial anatomical structures so that a superposition of them in acommon three-dimensional coordinate system may occur. In a lesspreferred embodiment, simple two dimensional data sets could be used, inwhich the 2 dimensional data sets are overlapped to create a virtualpatient in two dimensions. Examples of this might be using x-ray andphotographs and creating the virtual patient without use of 3D data.

The type of image data that will be obtained will vary depending on theavailable image acquisition devices available to the practitioner, andthe imaging techniques that are most pertinent for a given patient,given the totality of the circumstances. Preferably, the system employssoftware simulation of changes in shape or position of craniofacialstructures (e.g., teeth or jaw) on the visual appearance, e.g., smile,of the patient. Accordingly, at least one of the data sets will includenormally include data regarding the surface configuration of the faceand head. A commercially available digital CCD camera 28 (FIG. 1), e.g.,a color or black and white digital camera available from Sony or Canon,can be used to obtain this information. Preferably, the image data iscolor image data. The data sets are obtained by photographing thepatient's head and face at various viewing angles with the camera andstoring the resulting image files in the memory of the computer. Theseimages can provide a basis for creating a morphable face model.

The image data regarding the patient's exterior appearance can beobtained through other means including via scanning of the head and faceof the patient via the hand-held 3D-scanner 30 described in thepublished OraMetrix PCT application, publication no. WO 01/80761,incorporated by reference herein. If this approach is used, it may bebeneficial to apply a thin layer of non-toxic, opaque and reflectivesubstance to the skin prior to scanning to insure adequate data captureby the hand-held scanner. A suitable opaquing substance is described inthe patent application of Nancy Butcher et al. Ser. No. 10/099,042 filedMar. 14, 2002, entitled “Method for Wet-Field Scanning,” the contents ofwhich are incorporated by reference herein. In operation, the scannercaptures a sequence of overlapping images of the surface of the patientas the scanner is held by the hand and moved about the face. The set ofimages can be obtained in only a few minutes. Each image is converted toa set of X, Y and Z coordinate positions comprising a cloud of pointsrepresenting the surface of the face. The point clouds from each imageare registered to each other to find a best fit to the data. Theresulting registered point cloud is then stored in the memory as avirtual three-dimensional object. The construction, calibration andoperation of the scanner, and the manner of converting scanned data topoint clouds and registering three-dimensional point clouds to form athree-dimensional object is described at length in the published PCTapplication of OraMetrix WO 01/80761, and therefore omitted from thepresent discussion for the sake of brevity. Other types of scanners orcoordinate measuring instruments could be used in less preferredembodiments, such as the scanning devices in the Yamany et al. articlesreferenced previously.

Aside from surface data of the patient obtained by the camera 28 or 3Dscanner 30, the system typically will include the capture of additionaldata representing the teeth of the patient, and also capture ofadditional data representing craniofacial structures not visible to thenaked eye using other imaging devices 36 (FIG. 1). For example, thesystem will acquire digitized images from an X-ray machine capturingX-ray photographs of the patient's head, jaw, teeth, roots of teeth, andother craniofacial structures. These photographs are digitized andstored in the memory of the computer system. Video images can be used totrack functional movements such as movement of the jaws and smiling.

As other possible examples, three-dimensional magnetic resonance imagesof the patient's head or jaws are obtained and stored in the memory.Other examples include images acquired from a computed tomography (CT)scanner, ultrasound imager, or other type of imaging device.

While the above discussion has described how 3D image of the face can beobtained from a three-dimensional scanner, there are other possibilitiesthat may be used in the practice of alternative embodiments. One suchalternative is creating a 3D virtual face from a series of 2-D colorphotographs. This technique is known and described in Pighin et al.,Synthesizing Realistic Facial Expression from Photographs, ComputerGraphics Proceedings SIGGRAPH '98, pp. 78-94 (1998); Pighin et al.,Realistic Facial Animation Using Image-based 3D Morphing, TechnicalReport no. UW-CSE-97-01-03, University of Washington (May 9, 1997); andBlantz et al., A Morphable Model for The Synthesis of 3D Faces, ComputerGraphics Proceedings SIGGRAPH '99 (August, 1999), the contents of whichare incorporated by reference herein. Basically, in this alternative,two-dimensional color pictures of the face are taken which are convertedautomatically to a textured 3 dimensional model using a ‘morphablemodel’ technique. Here, the phrase “textured 3 dimensional model” isused in the particular sense of a colorized three-dimensional object,with the word “texture” synonymous with color data, as that term is usedin this particular art.

Morphable models can be built based on various known approaches such asoptic flow algorithms or active model matching strategy, or acombination of both. One approach is to scan a set of 2D faces. A shapevector containing 3D vertices and texture vector containing RGB colorvalues of each vertex represents the geometry of the face. Each face isdivided into sub regions such as eyes, nose, mouth etc. Blending thesub-regions at the borders generates the complete 3D face. Automaticmatching and estimating 3D face of a 2D color image from morphable modelis carried out as follows:

New Shape (Sn) and texture (Tn) are computed as follows:

(1) Sn=Sa+Σαs;

(2) Tn=Ta+Σβt, where Sa and Ta are the averages of Shape S and Texture Tover all the 3D face datasets; s & t are the eigenvectors of thecovariance matrices; α and β are the coefficients of the facial shapeand texture for all the faces, and n is a sub-region index.

Rendering parameters ρ contain camera position, object scale, imageplane rotation and translation and light intensity. From Bayes decisiontheory, the set of parameters, (α,β,ρ) are determined with maximumposterior probability for getting a corresponding 3D face from a 2Dimage.

Three-dimensional image data sets of the upper and lower archesincluding upper and lower teeth are preferably created with a 3D opticalscanner 30, such as the OraMetrix hand-held in-vivo scanner. If the 3Djaw model has no texture model, i.e., no color data, the texture datacan be extracted from the 2 dimensional colored picture of the upper andlower jaw and mapped to the 3D coordinates on the jaw model using acylindrical projection technique. In this technique, a map isconstructed in texture space, that for each point (u, v), specifies atriangle whose cylindrical projection covers that point. The 3D point pcorresponding to point (u, v) in texture space is computed byintersecting a ray with the surface of the corresponding point in the 2Dcolored image.

Superposition or Registration of the Data Sets

After the images of the face, craniofacial structures, X-rays, teethetc. are obtained and stored in memory in digital form they aresuperimposed on each other (i.e., registered to each other via softwarein the workstation) to create a complete virtual patient model on theworkstation. The superposition of the sets of image data may bedeveloped as an automatic software process, or one in which there isuser involvement to aid in the process. In one possible example, thethree-dimensional textured model of the face is properly aligned withthe 3D jaw model obtained from the intra-oral scan, 3D skull data fromCT scan, and 2 dimensional X-rays to create a virtual patient model. Forcorrect alignment of the data sets to each other, a preferred methodexecuted by the software selects three or more corresponding points onthe 3D jaw and the 3D face, and then computes a transformation matrix tore-orient the 3D face relative to the 3D jaw. This transformation matrixwill contain the information needed to rotate and translate the 3D facerelative to the 3D jaw in a best-fit manner to align the two to eachother. Methods of calculation of transformation matrices to achieveregistration are taught in the published PCT patent application ofOraMetrix, Inc., WO 01/80761, cited previously. Similar methods are usedfor registering the CT scan data and X-ray data to the combined 3D faceand jaw model. Once the superposition is achieved, the resulting modelis displayed on the workstation user interface. The user is providedwith tools for simulating movement or repositioning of craniofacialstructures of the virtual patient, and the computer animates suchmovement or repositioning and shows the effect of such movement orrepositioning on the external visual appearance of the patient.

An example of registering scan data of a jaw from an intra-oral scan toa 3D face model using human interaction is shown in FIGS. 2-7. FIG. 2 isa flow chart showing a software method of three-dimensional facecreation from scanning systems, which may be executed in software in thecomputer system 10 of FIG. 1. There are two possible approaches forcreating the 3D face, one using a color digital camera 28 (FIG. 1) andanother using scanning of the face using the hand held scanner 30 andscanning node 31 (FIG. 1), as one possible example. Other types ofscanning or imaging techniques can be used. In the situation in which acolor digital camera is used, at step 40 a set 24 of 2D digital colorphotographic images of the face and head are obtained and stored in thememory 22 of FIG. 1. The set 24 of images is supplied to a module 42which creates a virtual 3D face using an active model matching strategy,using the techniques known in the art and described in Pighin et al.,Synthesizing Realistic Facial Expression from Photographs, ComputerGraphics Proceedings SIGGRAPH '98, pp. 78-94 (1998); Pighin et al.,Realistic Facial Animation Using Image-based 3D Morphing, TechnicalReport no. UW-CSE-97-01-03, University of Washington (May 9, 1997); andBlantz et al., A Morphable Model for The Synthesis of 3D Faces, ComputerGraphics Proceedings SIGGRAPH '99 (August, 1999). This 3D face model isthen stored in the hard disk memory of the computer 10, as indicated atprocess module 44.

In alternative embodiments, a 3D scanning of the face using a laser or3D optical scanner is performed, as indicated at 44. The 3D model isprovided to a module 46 which creates a morphable model of the face andhead with an optic flow algorithm. This morphable model is provided tothe module 42 for creating a 3D face. At step 50, the software inquiresas to whether a morphable 3D face is available, and if not theprocessing of module 42 executes, in which a 3D morphable model of theface is created. If a morphable 3D face is already available, thesoftware inquires at step 54 as to whether texture (color) informationis available to add to the 3D face. (Note that in many 3D scannersystems there is no acquisition of color information, only spatialinformation). If color information is not available, the processingproceeds to module 56. In module 56, the color data is provided to the3D model to create a 3D color morphable virtual model. The color data issupplied from the digital photographs of the patient, obtained at step40. The texture information is supplied to the 3D model from the scannerusing a cylindrical projection technique in module 56 (or by using anyother known technique). The textured, morphable 3D model of the face andhead is stored as indicated at module 44.

An alternative software method or process for creating a 3D model of theface is shown in FIG. 3. The method involves the acquisition of a 3Dcolor face model 52 (using for example the techniques of FIG. 2), theacquisition of 3D color model of the teeth 54, and the acquisition of amodel 56 of the skull using a CT scanner. These three models aresupplied to a module 58 which performs an aligning transformation on thedata sets from each of these modules. The aligning transformationprocess 58 basically scales and provides the necessary X, Y and Ztranslations and rotations to place the data sets into a commoncoordinate system such that common anatomical structures overlap eachother. The complete 3D face model is stored as indicated at step 60 andthen supplied to an overlay transformation module 66. The overlaytransformation module 66 obtains a set of 2D color face photographs 62and X-Rays 64, and overlays them to the complete 3D face model to resultin a combined, composite model of the face, skull, teeth, and associatedtooth roots, bone and other anatomical data. This compositerepresentation of the patient is stored in a database 68 for the system100.

FIG. 4 shows a process that can be used to combine 3D scan data with 2Dcolor photographs to create a 3D color model of the teeth. In step 70,the teeth are scanned with the hand-held intra-oral scanner 30 ofFIG. 1. The resulting data represent a 3D model of the dentition, whichis stored in the computer 10. This process is described in the publishedPCT application of OraMetrix, Inc. cited previously. At step 72, 2Dcolor photographs of the teeth are obtained with a color digital camera.In one possible embodiment, the hand-held scanner 30 includes a videocamera that obtains a continuous stream of color video frames separateand apart from the acquisition of 3D image data. The color photographsof the dentition at step 72 could be obtained in this manner.

At step 74, a 3D textured model of the teeth is created using acylindrical projection technique. Basically, in this technique, thecolor data from the color photographs is projected onto the tooth data.The tooth data can be represented as triangular surfaces, with thevertices of each triangle being adjacent points in a point clouddefining the surface of the tooth. The color is projected on thesurfaces, and each surface is assigned a value associated with aparticular color. The result is a 3D color model of the teeth. FIG. 4further shows a step of simulating tooth movements and resulting changeson the appearance of the patient, either dynamically or statically. Thisstep can be done regardless of whether the user chooses to create colormodels of teeth or whether they use non-colored models of teeth.Intermediate steps which may need to be performed, such as separation ofa composite model of the entire arch into individual moveable virtualtooth objects, are not shown, but are known in the art and explained inthe OraMetrix published PCT application WO 01/80761.

FIGS. 4A-4E show several screen displays from a user interface of theunified workstation that illustrate the process of texture mapping a 3Dobject (here, teeth) by projection of color data from a 2D photograph.After a patient's dentition is scanned, the virtual teeth and gingivafor both upper and lower arches are represented as a single surface, inthe present example a triangle mesh surface. FIG. 4A shows a 2D digitalphotograph of teeth/gingivae 71 displayed in a graphical window 73 alongwith a 3D virtual model of the teeth 75 to one side. The 2D digitalphotograph 71 is scaled up or down in size as necessary to as to beapproximately the same in scale (size) as the 3D model of the teeth 75.This is accomplished using any suitable icons or mouse action, such asclicking on the 2D photograph and scrolling up or down with the mouse tochange the size of the 2D image so that it matches the size of the 3Dmodel. FIG. 4B shows the surface of the teeth and gingivae of the 3Dvirtual model 75 in greater detail. The surface of the model 75comprises a set of minute interconnecting triangle surfaces, with thevertices of the triangle surfaces being points that represent thesurface. This is only one possible format for representing the surfaceof a 3D object.

After the 2D photograph and 3D model have been scaled, a translation isperformed to as to overlap the 3D model and the 2D photograph. FIG. 4Cshows the 2D picture 71 transformed by scaling and translation such thatit is superimposed on the 3D model 75. This superposition could beperformed manually or automatically. For example, the user can click anddrag the 2D digital photograph 71 and manually move it using the mouseso that it overlaps exactly the 3D model 75. The color information inthe 2D photograph 71 is projected and mapped to the individual trianglesurfaces forming the lower jaw and upper jaw of the 3D model 75 using,for example, a projection algorithm. The result, a textured 3D model, isshown in FIG. 4D. FIG. 4E shows a textured 3D model after rotation onthe user interface.

Occlusal and lingual 2-D color photographs of each jaw are also obtainedand texture data is mapped to the surface data. The result is a completetrue color 3D model of the teeth of both arches.

FIG. 5 is an illustration of a screen display on the user interface ofthe computer 10. The display shows a 3D morphable model 102 of patienton the left hand side of the display, in a given arbitrary coordinatesystem X, Y, Z. The morphable model 102 is obtained, for example, fromcolor photographs using the techniques described previously. Athree-dimensional model 104 of teeth of the patient is shown on theright hand side of the screen. The 3D model of the teeth 104 can beobtained from intra-oral scanning using the scanner 30 of FIG. 1, from alaser scan of a physical model of the dentition obtained from animpression, from a coordinate measuring device or some other source. Thesource is not particularly important. The 3D model of the teeth 104 isshown in a separate coordinate system X′, Y′, Z′. Screen displayincludes various icons 35 the allow the user to position the tooth model104 relative to the morphable model 102 in order to combine the two in acommon coordinate system and construct a composite model.

In FIG. 6, the user has activated an “Align References” icon, whichcauses the screen display to show the box 106 on the left hand side ofthe screen. The user is provided with the option to pick points thatrepresent anatomical structures that are common to both the morphablemodel 102 and the 3D tooth model 104. In this particular situation, theuser has selected with the mouse two points on the lower arches whichlie at the intersection of the teeth and the gums. These two points areshown as a triangle 108 and a square 110. Obviously, other points couldbe chosen. The user then clicks on the “Apply” tab 112. The result isshown in FIG. 7, in which the 3D tooth model 104 is combined with themorphable face 102 model to produce a combined virtual patient model 34.

In the example of FIGS. 5-7, the morphable model 102 was already scaledto the same scale as the tooth model 104. In other words, the datarepresenting the morphable face model indicates that the spatialdimensions of the teeth in the morphable face model is substantially thesame as the spatial dimensions of the virtual tooth model 104. Methodsof performing scaling are described below.

FIG. 8 is an illustration of an alternative embodiment of a virtualpatient model 34. In this embodiment, the model 34 is a combination ofdata 102 representing a morphable face, obtained from a plurality of 2Dcolor photographs, and skull data 114 obtained from a CT scan. The twosets of data are shown in a common coordinate system, appropriatelyscaled. The manner of combining the two data sets can be using theapproach described in FIGS. 6 and 7. Alternatively, the user could clickand drag using a mouse one or the other of the data sets and manuallyposition it until it is in the correct position. As yet anotheralternative, the software could be programmed to find common overlappingfeatures (such as for example teeth) using surface matching algorithms,and then position the CT scan model relative to the face model such thatthe common features overlap exactly.

FIG. 9 is a screen shot of yet another possible embodiment of a virtualpatient model. This model combines face data 102 from a morphable facemodel (typically obtained from 2D color photographs or a 3D scannedimage of the face), skull data 114 from a 2D or 3D X-ray, and tooth data116 obtained from a 2D or 3D image of the teeth, such as for example andin vivo scan of the patient or a scan of a model of the teeth. Themanner of creating the virtual patient model can be for example usingthe procedure of FIG. 3 and FIG. 6-7. The morphable face model isaligned relative to the X-ray scan data either automatically or usingsome human interaction. The 2D X-Ray data can be morphed into 3D digitaldata using the morphable model algorithms cited previously.Alternatively, the 2D X-Ray data can be combined with 3D optical scandata of crowns of the teeth to create a combined X-Ray/3D tooth model,which is then combined with the X-ray/morphable face model. This processmay be optimized by using virtual template tooth roots, which aremodified to fit the X-ray data, and then this combined 3D root model iscombined with crown data to produce a complete set of 3D teeth,including roots. This combined model is merged into the X-ray/morphableface model using the techniques of FIGS. 6 and 7 (selecting commonpoints then using the “Apply Points” icon, FIG. 7, item 112), usingclick and drag techniques, or any other appropriate registration ortransformation technique.

Once the virtual model is created, the user is provided with tools thatallow the user to hide one or more image data in order to study certainfeatures. Furthermore, the user is provided with navigation tools withthe ability to manipulate the model so as to view it from anyuser-specified perspective. For example, in FIG. 10 a screen shot isshown of the superposition of skull data 114 with tooth data 116. Inthis example, complete 3D models of the teeth 116 are created from anin-vivo scan or scan of a model, or from some other source of scanning.Alternatively, the complete 3D tooth models 116 are created fromcombining X-Ray data with 3D models of teeth obtained by a scan of thecrowns of the teeth (using the scanner 30 of FIG. 1 or from a laser scanof a physical model of the dentition), and/or with the use of templatetooth roots that are modified to match the X-ray data.

Scaling of Data

When digital image data from multiple sources are combined orsuperimposed relative to each other to create a composite model, it maybe necessary to scale data from one set to the other in order to createa single composite model in a single coordinate system in which theanatomical data from both sets have the same dimensions inthree-dimensional space. Hence, some scaling may be required. Thissection describes some approaches to scaling that may be performed inone possible embodiment of the invention.

FIG. 11A-11E are views of scan data 200 representing the front, rightside, left side, lower and upper arches of a patient. The data includesthe teeth 202 and the gingiva 204. FIG. 12 illustrates a technique ofscaling the orthodontic data to match the actual orthodontic size.Depending on of the scanning technique, the orthodontic data will notcompletely reproduce the exact size of the teeth and other portions ofthe orthodontic structure. To facilitate the accuracy of thethree-dimensional digital model, at least one tooth 206 can be markedutilizing one or more markings 208. The marking is done prior toobtaining the orthodontic data. Once the orthodontic data for the tooth206 is obtained, scaling reference points 210 are also obtained. Thescaling reference points are the points in the scan data that representthe image of the markings 208. A determination between the differencesbetween the scaling reference points 210 and the actual markings 208determine a scaling factor 212. As one of average skill in the art willreadily appreciate, having the actual markings 208 and the scalingreference points 210, a variety of mathematical operations may be usedto determine the scaling factor 212. For example, the coordinatedifferences (distance) between each of the vertices of the triangle maybe utilized. As one of average skill in the art will further appreciate,a different number of markings 208 may be utilized. For example, twomarkings may be used or four markings may be used, etc. In addition,more than one tooth may be marked with similar markings 208. Note thatthe markings may be on the exterior of the patient, and a localtriangulation technique may be used to obtain the scaling factor.Further note that the scaling factor 212 determination is based on anassumption that the scan data will have a linear error term in each ofthe x, y and z axis, such that a single scaling factor is determined andused to scale each of the teeth as well as the other aspects of theorthodontic structure of the patient.

FIG. 13 illustrates an alternate marking technique for determining ascaling factor for the orthodontic data. As shown, an actual tooth 206is marked with a marking 208. The marking 34 is of a substantial size soas to be adequately measured. Once the orthodontic data is obtained, theorthodontic data of the tooth 202 and a corresponding scaling referencepoint (area) 210 are used to determine the scaling factor 212. As one ofaverage skill in the art will readily appreciate, a simple mathematicalfunction may be used to determine the scaling factor 212 based on thesize (diameter) difference between the actual marking 34 and the scalingreference point 36. As an alternative to marking as described withreference to FIGS. 12 and 13, the actual tooth size, and the size of themodel of the tooth, may be measured and used to determine the scalingfactor. Accordingly, the difference between the actual tooth size thesize of the tooth in the scan data will constitute the scaling factor.

When three-dimensional scanning of the type described in the publishedPCT application or OraMetrix is used, scaling of the three-dimensionaldata is not needed as a true, accurate and to scale three-dimensionalimage is obtained through the use of triangulation. Likewise, a truethree-dimensional image can be obtained techniques such as computedtomography. However, for video or photographic data, and for X-ray data,scaling such as shown in FIGS. 12 and 13 may be needed in order to mergethat data to other data such as 3D scan data.

FIG. 14 illustrates a two-dimensional representation of image data, suchas a graphical diagram of a radiographic image, such as an x-ray of afew teeth. In another embodiment, the radiographic image can be acomputed tomographic image volume. As previously mentioned, theorthodontic data contains three-dimensional images of the surface of theorthodontic structure. X-rays provide a more detailed view of the teethand surrounding hard and soft tissue as two dimensional image data. Asshown in FIG. 14, each tooth 206 includes a crown 220, and a root 222and is embedded in bone 224. Accordingly, the orthodontic data 200 ofFIG. 11 only illustrates the crown 206 of the teeth. As such, a completethree-dimensional model of the orthodontic patient requires the rootsand bone to be included.

FIG. 15 illustrates a technique of using the scaled digital model 226 ofthe tooth's crown to produce an integrated or composite digital model228 of the tooth. In this embodiment, the x-rayed data 230 of the toothis used in comparison with the scaled digital model 226 to determine aper tooth scaling factor. The scaled digital model 226 of the tooth ispositioned to be planar with the x-ray of the tooth 230. Having obtainedthe proper orientation between the two objects, the per tooth scalingfactor is determined and subsequently used to generate the compositescaled digital model 228 of the tooth. In a specific embodiment, the pertooth scaling factor is required for current x-ray technologies, sincex-rays produce a varying amount of distortion from tooth to toothdepending on the distance of the tooth from the film, the angle of x-raytransmission, etc.

To more accurately map the two-dimensional images of a tooth onto thethree-dimensional model, multiple angles of the tooth should be used.Accordingly, a side, a front, and a bottom view of the tooth should betaken and mapped to the scaled digital model of the tooth. Note that thebone and other portions of the orthodontic structure are scaled in asimilar manner. Further note that MRI images, and any other imagesobtained of the orthodontic patient, may also be scaled in a similarmanner. A more complete representation of the tooth roots may beobtained using standardized, template 3D virtual tooth roots, applyingthe X-Ray data to the template tooth roots and modifying their shapeaccordingly, and them applying the modified template tooth root to thescan data of the crown to create a scaled, complete virtual tooth objectincluding tooth roots.

FIGS. 16A-16C illustrate a graphical diagram of selecting orientationreference points based on physical attributes of the orthodonticstructure. The orientation reference points 262 and 266 will besubsequently used to map the digital image of the orthodontic structureinto a three-dimensional coordinate system that will not change duringthe course of treatment. In this example, the frenum 264 has beenselected to be one of the orientation reference points 266 and the rugae260 has been selected as the other reference point 262. The frenum 264is a fixed point in the orthodontic patient that will not change, orchange minimally, during the course of treatment. As shown, the frenum264 is a triangular shaped tissue in the upper-portion of the gum of theupper-arch. The rugae 260 is a cavity in the roof of the mouth 268 inthe upper-arch. The rugae will also not change its physical positionthrough treatment. As such, the frenum 264 and the rugae 260 are fixedphysical points in the orthodontic patient that will not change duringtreatment. As such, by utilizing these as the orientation referencepoints 262 and 266, a three-dimensional coordinate system may be mappedthereto. Note that other physical attributes of the orthodontic patientmay be used as the orientation reference points 262 and 266. However,such physical points need to remain constant throughout treatment.Accordingly, alternate physical points include the incisive papilla,cupid's bow, the inter-pupillar midpoint, inter-comissural midpoint(e.g., between the lips), inter-alar midpoint (e.g., between the sidesof the nose), the prone nasale (e.g., the tip of the nose), sub-nasale(e.g., junction of the nose and the lip), a dental mid-line point, apoint on the bone, a fixed bone marker such as an implant (e.g., a screwfrom a root canal, oral surgery).

The x, y, z coordinate system may be mapped to the physical points onthe digital model of the orthodontic structure in a variety of ways. Inone example, the origin of the x, y, z coordinate system may be placedat the frenum 264, the z-axis aligned with reference to the frenum andthe rugae 260, and the x-axis is aligned with the midline of the upperand/or lower arch. This is further illustrated in FIGS. 17 and 18. Notethat an external positioning system may be used to obtain theorientation reference points. For example, the patient may sit in achair at a specific location of an examination room that includes atriangulation positioning system therein. As such, when the patient isscanned, the scanned images may be referenced with respect to the room'striangulation positioning system.

FIG. 17 illustrates a graphical representation of mapping theorientation reference points 262 and 266 to the x-z plane of thethree-dimensional x, y, z coordinate system. In this illustration,orientation point 266, which corresponds to the frenum 264, is selectedas the origin of the x, y, z coordinate system. Note that any locationmay be selected as the origin 72. The orientation points 262 and 266 areused to determine an x, z plane orientation angle 262. Typically, the x,y, z coordinate system will be selected such that when looking at thepatient from a frontal view, the x direction will be to right of thepatient, the y direction towards the top of the patient's head and the zdirection will be out away from the patient. As one of average skill inthe art will appreciate, the orientation of the x, y, z plane may be inany orientation with respect to the reference points 262 and 266.

The x-y plane is mapped to the orientation reference point 262 and 266as shown in FIG. 18. The orientation reference point 262 and 266 areused to generate an x-y plane orientation angle 284. Based on the x-yplane orientation angle 284 and the x-z plane orientation angle 262, adigital model of a tooth 270 may be positioned in three-dimensionalspace with respect to the x, y, z coordinate system. As shown in FIGS.17 and 18, the digital model of the tooth 270 includes a tooth depth278, an angle of rotation 276 with respect to the x-z axis, an angle ofrotation 282 with respect to the x-y plane, a positioning vector 274which is in a three-dimensional space, the length of the tooth includingthe crown dimension, and the root dimension. Accordingly, each tooth isthen mapped into the x, y, z coordinate system based on the tooth'scenter, or any other point of the tooth, and the dimensions of thedigital model of the corresponding tooth. Once each tooth has beenplaced into the x, y, z coordinate system, the digital model of thetooth is complete. Note that the lower-arch is also referenced to the x,y, z coordinate system wherein the determination is made based on theocclusal plane of the patient's orthodontic structure. Alternatively,the lower-arch may include a separate three-dimensional coordinatesystem that is mapped to the coordinate system of the upper-arch. Inthis latter example, fixed points within the lower-arch would need to bedetermined to produce the lower arch's three-dimensional coordinatesystem.

Treatment Planning

The computer or workstation 10 (FIG. 1) that includes the software forgenerating the patient model preferably includes interactive treatmentplanning software that allows the user to simulate various possibletreatments for the patient on the workstation and visualize the resultsof proposed treatments on the user interface by seeing their effect onthe visual appearance of the patient, especially their smile. Theinteractive treatment planning preferably provides suitable tools andicons that allow the user to vary parameters affecting the patient. Suchparameters would include parameters that can be changed so as tosimulate change in the age of the patient, and parameters that allow theuser to adjust the color, texture, position and orientation of theteeth, individually and as a group. The user manipulates the tools forthese parameters and thereby generates various virtual patient modelswith different features and smiles. The patient models are displayed onthe user interface of the workstation where they can be shared with thepatient directly. Alternatively, the workstation can be coupled to acolor printer. The user would simply print out hard copies of the screenshots showing the virtual patient model. Additional features related totreatment planning are disclosed in the patent application of RohitSachdeva filed concurrently, entitled METHOD AND SYSTEM FOR INTEGRATEDORTHODONTIC TREATMENT PLANNING USING UNIFIED WORKSTATION, Ser. No.10/428,461.

The manner in which the software is written to provide tools allowingfor simulation of various parameters can vary widely and is notconsidered especially critical. One possibility is a Windows-basedsystem in which a series of icons are displayed, each icon associatedwith a parameter. The user clicks on the icon, and a set of windows aredisplayed allowing the user to enter new information directing a changein some aspect of the model. The tools could also include slide bars, orother features that are displayed to the user and tied to specificfeatures of the patient's anatomy. Treatment planning icons for movingteeth are disclosed in the published PCT application of OraMetrix, Inc.,publication no. WO 01/80761 and cited previously, which gives some ideaof the types of icons and graphical user interface tools that could beused directly or adapted to simulate various parameters. Otherpossibilities are disclosed in the patent application of Rohit Sachdevafiled concurrently, entitled METHOD AND SYSTEM FOR INTEGRATEDORTHODONTIC TREATMENT PLANNING USING UNIFIED WORKSTATION, Ser. No.10/428,461.

Once the user has modified the virtual patient model to achieve thepatient's desired feature and smile, it is possible to automaticallyback-solve for the teeth, jaw and skull movement or correction necessaryto achieve this result. In particular, the tooth movement necessary canbe determined by isolating the teeth in the virtual patient model,treating this tooth finish position as the final position in theinteractive treatment planning described in the published OraMetrix PCTapplication, designing the bracket placement and virtual arch wirenecessary to move teeth to that position, and then fabricating the wireand bracket placement trays, templates or jigs to correctly place thebrackets at the desired location. The desired jaw movement can bedetermined by comparing the jaw position in the virtual patient model'sfinish position with the jaw position in the virtual patient model inthe original condition, and using various implant devices or surgicaltechniques to change the shape or position of the jaw to achieve thedesired position.

The virtual patient model as described herein provides a common set ofdata that is useable in a variety of orthodontic or other treatmentregimes. For example, the initial and final (target) digital data setsof the patient's tooth positions can be relayed to a manufacturer ofcustomized transparent removable aligning shells for manufacture of aseries of aligning devices, as taught in the Chisti et al. patents citedpreviously. Alternatively, the tooth positions may be used to derivecustomized bracket prescriptions for use with a flat planar archwire orother non-planar archwire. Furthermore, surgical devices such assurgical archwires, splints, prosthetic devices, and restorative devicescan be fabricated with these data sets. Methods of fabricated customizedarchwires from data sets indicating bracket position and tooth geometryare disclosed in the patent application of Werner Butscher et al., Ser.No. 09/834,967. allowed, which is incorporated by reference herein.Methods of fabricating bracket placement jigs are described in U.S.patent application Ser. No. 09/560,127, allowed, the contents of whichare incorporated by reference herein.

The choice of which treatment modality, and whether to use anyadditional treatment or therapeutic approaches (including surgery) willdepend on the patient in consultation with the treating physician. Theintegrated environment proposed herein provides essentially a platformfor a variety of possible treatment regimes. Further, the creation anddisplay of the virtual patient model provides for new opportunities inpatient diagnosis and sharing of patient information across multiplespecialties in real time over communications networks.

FIG. 19 is a block diagram of an integrated workstation environment forcreation of the virtual patient model and diagnosis, treatment planningand delivery of therapeutics. The workstation environment shown in blockdiagram form in FIG. 19 may incorporate many of the hardware aspectsshown in FIG. 1, including scanning or imaging devices 28/36 forcapturing two dimensional images, such as a color digital camera orX-Ray machine. The workstation environment will preferably includescanning or imaging devices 30/36 for capturing three dimensional imagesand creating 3D models of the patient, including one or more of thefollowing: laser scanners for scanning a plaster model of the teeth,optical scanner such as the OraMetrix hand-held scanner 30 referenced inFIG. 1, CT scanner or MRI. In some instances, the scanning devices maybe located at other facilities, in which case the 3D scans are obtainedat another location and the 3D data is supplied to the workstation 10(FIG. 1) over a suitable communications channel (Internet) or via a disksent in the mail.

The workstation includes a memory storing machine readable instructionscomprising an integrated treatment planning and model manipulationsoftware program indicated generally at 300. The treatment planninginstructions 300 will be described in further detail below. Thetreatment planning software uses additional software modules. A patienthistory module 302 contains user interface screens and appropriateprompts to obtain and record a complete patient medical and dentalhistory, along with pertinent demographic data for the patient.

A module 304 contains instructions for designing custom dental andorthodontic appliances. These appliances include both fixed appliances,e.g., brackets, bands, archwires, crowns and bridges, as well asremovable appliances including aligning shells, retainers and partial orfull dentures. In one possible embodiment, the module 304 may be locatedand executed at the site of a vendor of custom orthodontic applicants.The vendor would receive an order for a custom appliance specifically tofit an individual patient. Module 34 would process this order andcontaining instruction for designing the appliance to fit the individualmorphology and condition of the patient. The vendor would take theappliance design, manufacture the appliance in accordance with thedesign, and then ship the custom appliance to the practitioner. Examplesof how the appliance design module 304 might be implemented include theappliance design software developed by OraMetrix and described in thepublished PCT patent application cited previously, the customizedbracket, jig and wire appliance design software of Ormco described inthe issued Andreiko patents (see, e.g., U.S. Pat. No. 5,431,562) and inthe published patent application of Chapoulaud, US patent publicationno. 2002/002841, the techniques of Chisti et al., U.S. Pat. Nos.6,227,850 and 6,217,325, all incorporated by reference herein.

The treatment planning software 300 also obtains information on standard(“off the shelf”) dental or appliances from a module 306, which storesmanufacturer catalogs of such appliances, including 3D virtual models ofthe individual appliances.

The treatment planning software includes a module 308 that allows theuser to input selections as to variable parameters that affect thevisual appearance of the patient, as input to a craniofacial analysismodule 328 described below. The variable parameters include patientfactors: age, weight, sex, facial attributes (smile, frown, etc.). Thevariable parameters also include parameters affecting the teeth,including texture (color), position, spacing, occlusion, etc. Thevariable parameters further include various illumination parameters,including reflectivity of the skin, ambient light intensity, and lightdirection. These parameters are accessed though appropriate icons on thescreen display, such as the icons shown in FIGS. 4-7, and pop-updisplays that appear that prompt the user to enter or vary the selectedvariable parameter.

The treatment planning software further uses a diagnosis and simulationmodule 310 that displays diagnosis data graphically and/or in reportformat. This diagnosis data includes teeth position, 3D face and smileappearance, and various facial attributes.

The software further includes third party practice management software312. Information about treatment plans generated by the craniofacialanalysis module 328 is input to the practice management software 312.Based on the treatment plan, this software generates the most productivescheduling of appointments for the patient. The practice managementsoftware 312 also generates reports, provides insurance and benefittracking, and supports electronic claims filing with the patient'sinsurance company. Preferably, the practice management software providesa flexible scheduling of patient appointments based on progress oftreatment of the patient's craniofacial anatomical structures. Theprogress of treatment is obtained from periodically obtaining updatedthree-dimensional information regarding the progress of treatment of thecraniofacial features of the patient. For example, the patient isperiodically rescanned during the course of treatment. A new virtualpatient model is created. Depending on the progress of treatment (e.g.,movement of the teeth to target positions) the patient may be scheduledfor more or less frequent visits depending on their progress.

Referring again generally to the treatment planning software 300, thesoftware includes a 3D model generation module 314 that uses as inputthe 2D and 3D scanning devices. A 3D virtual model of the patient iscreated by module 314, for example, in the manner described previouslyin FIGS. 2 and 3.

The system further includes a custom appliance management module 315.This module provides appliance specifications and 3D geometry data tothe vendor site for the purpose of providing necessary input for thedesign and manufacture of custom appliances, such as custom orthodonticappliances, for the patient. This module also provides updates to anappliance data module 324 for storing custom appliance data within thedatabase. The module 324 is responsible for managing the database of allthe appliances, including custom appliances.

The 3D virtual patient model is supplied to a knowledge database 316.The knowledge database includes a 3D Geometry data file 316 that storesthe 3D geometry data of the virtual patient model. This data is suppliedto a tagging module 322 and a morphable model module 320. The morphablemodel module 320 includes instructions for creating a morphable modelfrom various 3D model samples, using the techniques for example setforth in the article of Blantz et al., A Morphable Model for TheSynthesis of 3D Faces, Computer Graphics Proceedings SIGGRAPH '99(August, 1999).

The tagging module 322 includes instructions for tagging or placingpieces of information regarding the virtual patient model into eachpatient record, which is used for statistical procedures. In particular,the tagged information is supplied to a meta-analysis module 326. Themeta-analysis module implements a set of statistical procedures designedto accumulate experimental and correlational results across independentstudies that address a related set of research questions. Meta-analysisuses the summary statistics from individual studies as the data points.A key assumption of this analysis is that each study provides adifferent estimate of the underlying relationship. By accumulatingresults across studies, one can gain a more accurate representation ofthe relation than is provided by the individual study estimators. In oneexample, the software will use previous patient cases/studies to help inthe craniofacial analysis module 328. For example, surgery cases for“lip and chin” will be one set of independent studies, whereas jawsurgery to correctly position the upper and lower jaw will be another.Another example is pathology exhibited by the patient to drive thetreatment plan through meta-analysis. An orthodontist trying to alignthe upper and lower jaw will do a meta-analysis with the module 326 inorder to see how this treatment will affect the patient's lip and chin.

The output of the morphable model from module 320 and the meta-analysisfrom module 326 is provided to a craniofacial analysis module 328. Thismodule takes as input, patient information and the patient 3D virtualmodel to generate diagnosis and simulation data. Based on one or moresimulation results, this module 328, and/or module 330 generates atreatment plan and appliance selection. User involvement is contemplatedin modules 328 and 330. In particular, the user may interact with thepatient information and the morphable model, and vary the parameters308, to simulate different possible treatments and outcomes to arrive ata final or target treatment objective for the patient. The craniofacialanalysis module 328 may include some or all of the treatment planningfeatures described at length in the published PCT application ofOraMetrix, Inc. cited previously.

The software instructions included in the craniofacial analysis module326 preferably includes a set of instructions providing the user withuser interface tools (e.g., icons), for visually studying on the userinterface 16 the interaction of the craniofacial anatomical structuresand their relationship to the external, visual appearance of thepatient. For example, tools may provide a chewing simulation.Alternatively, the tools may provide a smile function in which the faceis morphed to smile, showing the position of the teeth, gums, lips andother structures. These tools simulate changes in the anatomicalposition or shape of craniofacial anatomical structures (teeth, lips,skin, etc.) and show the effect of such changes on the visual appearanceof the patient. As another example, the tools may include tools formodifying the shape or position of one or more bones of the upper andlower jaws, and show how those modifications affect the patient'sappearance and smile.

With reference to FIG. 7, the user would activate one of the icons 35 atthe top of the screen display. The icon may be associated with afunction that would allow the user to reposition the location of theupper and lower teeth. After the user changes the position of the teeth,the user would activate another icon, “smile”, and the face would morphto a smile with the teeth in the new position.

After the patient simulations have been completed and the patient andphysician are satisfied, the resulting data set of teeth position, jawposition, etc. are stored by the diagnosis and simulation module 310 ofFIG. 19. This module 310 preferably includes a routine for storing athree-dimensional representation of said patient's craniofacialstructures (e.g., teeth) in a format suitable for use by a manufacturerof orthodontic appliances. Each manufacturer may have a unique formatneeded for use by the manufacturer, and the routine takes that intoconsideration in storing the data. For example, a manufacturer mayrequire 3D digital models of the teeth in initial and final positions inthe form of triangle surfaces, along with archwire and bracketprescription data.

It is contemplated that the creation and usage of the virtual model mayoccur at the patient care site. In particular, the treating practitionerwill access the scan and photographic data, create the virtual modeltherefrom, and perform the treatment planning and simulation describedherein in their own office. Once the treatment plan is arrived at, thetreating physician can export the virtual patient model or some subsetof data to appliance manufacturers or specialists, as indicated in FIG.1.

Alternatively, the virtual patient model may be created at a remotelocation. In this latter example, a third party, such as an appliancemanufacturer, may be the entity that creates the virtual patient modeland makes it available to the treating physician. In this example, thetreating physician will have access to the scanners, X-Ray, digitalcamera, or other imaging device, obtain the required data from thepatient, and forward such data to the third party. The third partyexecutes the instructions to create, visualize and manipulate thevirtual patient model. This model can be transmitted to the treatingphysician for their review and usage. Then, either the third party couldcreate a proposed treatment for review and approval by the treatingphysician, or the treating physician could create the treatment plan.The plan is then transmitted to one or more appliance manufacturers forfabrication of therapeutic devices (e.g., brackets and wires, aligningshells, maxillary expansion devices, etc.)

A treatment plan created from the virtual patient model described hereinmay be one in which only one type of appliances, e.g. fixed ofremovable, is used during the entire course of the treatment. Forexample, the treatment plan may be one in which brackets and wires arethe type of appliance that is used. Or, alternatively, the treatmentplan may be one in which removable aligning shells are the type ofappliance that is used.

On the other hand, the treatment plan might be such that it is a hybridplan requiring the use of different types of appliances during thecourse of the treatment. In the hybrid orthodontic treatment plan, avariety of scenarios are possible. In one type of hybrid treatment plan,different types of appliances might be used at different times duringthe course of the treatment. For example, patient may start out withbrackets and wires and shift at some point during treatment to anapproach based on removable aligning shells. In another type of hybridtreatment plan, different types of appliances might be usedsimultaneously, for example in different portions of the mouth, forexample brackets and wires could be used for certain teeth andtransparent aligning shells uses for a different set of teeth. A hybridtreatment plan may be chosen right from the beginning, or it may beintroduced dynamically at any stage during the treatment course.

To develop a hybrid treatment plan, the treatment planning software willpreferably include features of the appliance design and treatmentplanning software of the manufacturers of the appliances that are usedin the hybrid treatment. As one example, the treatment planning softwaremay include the wire and bracket features of the OraMetrix treatmentplanning software described in the published application WO 01/80761, aswell as the treatment planning software described in the AlignTechnologies patents to Chisti et al., U.S. Pat. Nos. 5,975,893 and6,227,850.

The software would thus allow the user to simulate treatment withbrackets and wires for part of the tooth movement to reach a particularmilestone, and also design the configuration of intermediate toothpositions and configuration of removable aligning shells for theremainder of tooth movement. Alternatively, the shape of the aligningshells could be determined automatically via the treatment planningsoftware from the tooth configuration at which the shells are firstintroduced to the patient and the final tooth position in accordancewith the teachings of the Chisti et al. patents.

FIG. 20 is an illustration of the integration of the patient dataacquisition, treatment planning and appliance design functions that arefacilitated by a preferred embodiment of the unified workstation 14. Theworkstation is provided with a plurality of image data sets 400, whichcan include 2D data (e.g., photographs) 402, 3D image data 404 fromvarious 3D image sources, static models 406 of all or part of thepatient's craniofacial anatomy, dynamic models 408 of all or part of thepatient's craniofacial anatomy, color models 410, and possibly othertypes of image data. The workstation 14 includes software 314 (such asdescribed above in conjunction with FIG. 19) that takes any possiblecombination of this image data to produce a virtual patient model 34.From this virtual patient model, the workstation in one possibleembodiment includes one or more treatment planning tools or software 300for planning treatment for the patient. These treatment planning toolscould include specific software provided by vendors of treatmentplanning software or appliances, such manufacturer #1 software 412,manufacturer #2 software 414, software for manufacturers nos. 3, 4, 5,6, at 416, 418, 420, as shown. Such software would be operable on thevirtual patient model 34 as described at length herein. To provideinteroperability of the software on the virtual patient model, thevirtual patient model may have to have representations of the data thatis compatible with the software of various vendors, which is within theability of persons skilled in this art. Moreover, once appliance designshave been created by the various species of treatment planning software,the preferred embodiment of the workstation allows export of appliancedesign, tooth position data or other required outputs to any appliancemanufacturer so as to allow the manufacture of a customized orthodonticappliance. In other words, if the workstation is equipped with OraMetrixtreatment planning software, such software could output tooth positiondata, appliance design data and any other required data into a formatcompatible with the manufacturing requirements of any appliancemanufacture. This interoperability of data formats for appliance designis shown by arrows 421. Thus, the workstation provides a conversion orformatting of appliance design data into a data set or output formatspecified by any one of a variety of particular appliance manufacturers.In the illustrated embodiment, the available therapeutics data sets areshown as manufacturer no. 1 data set 422 (brackets and customizedwires), manufacturer no. 2 data set 426 (brackets and wires),manufacturer no. 3 data set 426 (removable aligning shells),manufacturer no. 4 data set 428 (brackets and wires), or still othersets 430. The appliance design data set is then furnished over theInternet to the vendor of such appliances for manufacture of a customappliance. Hybrid treatment plans, as described above, are onepossibility of a treatment plan that may be developed using theworkstation and virtual patient model described herein.

In FIGS. 21 and 22, the user has activated the icons across the top of adisplay on the workstation to simultaneously display both a twodimensional panoramic X-ray 501 of the teeth and jaw as well as the 3Dmodel 503 of the teeth, but with the teeth models spread out orflattened and represented in two dimensions, in approximate registrywith the panorama X-ray. The teeth models 503 that are shown in FIG. 21represent the tooth positions in a proposed treatment. This view allowsthe user to judge the position of the crowns of the teeth in theproposed position relative to the position of the roots in the bone, andthereby better ascertain whether the proposed treatment is appropriateIn FIG. 22, the user has unchecked the X-ray icon 505 and thus only the3D teeth are displayed. The superposition of 3D teeth, on a 2D image ofthe tooth and tooth roots, as shown in FIG. 21, could be done withvarious different X-ray views, including biplane, lateral, lateraloblique, panorama, etc., or even 3D images which show tooth roots andassociated bone structure, including CT scan images. The superpositionof the 3D crowns over the tooth roots, in two or three dimensions, givesthe user a new and improved tool for assessing the condition of thepatient and planning treatment.

In one possible variant of the invention, the treatment planningsoftware tools 300 are also provided at a remote location and some ofthe tasks of appliance design may be performed as a service by aseparate workstation, such as a workstation of an appliancemanufacturer. In this situation, the virtual patient model 34 could beprovided to the appliance manufacturer, a proposed treatment plan isprepared and furnished to the practitioner, and after the plan isapproved, the appliance manufacturer coordinates the furnishing ofappliance design data to any designated appliance manufacturers that areused to furnish the custom appliance.

In one possible embodiment, the treatment planning software 300 includesa set of instructions that perform a measurement function to measuredistances in two or three dimensions in the virtual patient model, e.g.,arch form shape measurements, and compare the measurements withreference dimensions for an “average” patient of similar age, sex, andrace. These average or “normal” measurements could be obtained in anyconvenient manner, for example from textbooks, organizations,practitioners, etc. These measurement tools would be invoked during thecourse of treatment to compare tooth movement and current tooth positionwith expected positions and if deviations occur, the variances could beused as information to modify one or more aspects of the treatment plan,such as change the appliance design.

Presently preferred and alternative embodiments of the invention havebeen set forth. Variation from the preferred and alternative embodimentsmay be made without departure from the scope and spirit of thisinvention. Furthermore, the reference in the claims to an opticalscanner for scanning the dentition of the patient is intended toencompass both an in-vivo scanner scanning the teeth of the patientdirectly or the use of an optical, laser, destructive, or other type ofscanner scanning a physical model of the teeth of the patient or animpression thereof.

Furthermore, many of the simulations involving the virtual patient modeldescribed herein that can be performed on the workstation, such as theage of the patient, the facial expression (smile, grimace, etc), thechange of position of anatomical structures, etc., can be eitherperformed as pure simulations, in which the end result is not known inadvance but an intermediate change of position of a anatomical componentis provided to effectuate the simulation, or as a morphing process inwhich the end result may be known but the intermediate steps are notknown. Hence, the terms “simulation” or “simulating” in the claims areintended to encompass both pure simulations as well as morphing typeoperations.

We claim:
 1. A method for diagnosis and planning treatment of a humanpatient using a computer, comprising the steps of: obtaining a first setof digital data representing patient craniofacial image information froma first imaging device; obtaining a second set of digital datarepresenting patient craniofacial image information from a second imagedevice different from said first image device; wherein said first andsecond sets of data representing at least in part common craniofacialanatomical structures of said patient, at least one of said first andsecond sets of digital data including data representing the externalvisual appearance or surface configuration of the face of the patient;wherein said first and second digital data sets are each obtained atdifferent points in time and are not captured in a correlated fashion;automatically, and/or with the aid of operator interaction,superimposing said first set of digital data and said second set ofdigital data so as to provide a composite, combined digitalrepresentation of said craniofacial anatomical structures created fromsaid first and second digital data sets each obtained at differentpoints in time and not captured in a correlated fashion in a commoncoordinate method; creating a virtual 3D face at least from a portion ofsaid craniofacial anatomical structures using an active model matchingstrategy; wherein said craniofacial anatomical structures include upperand lower jaws and teeth of said patient; wherein said teeth arepositioned relative to each other in malocclusion position of saidpatient; displaying said composite, combined digital representation ofsaid craniofacial anatomical structures, including said virtual 3D face,said upper and lower jaws and said teeth of said patient on a screendisplay of said computer using a user interface of said computer; makingmeasurements of said craniofacial anatomical structures usingmeasurement tools; and performing interactive treatment planning stepsusing said computer for planning orthodontic treatment of said patient.2. The method of claim 1, wherein said composite, combined digitalrepresentation comprises a three-dimensional representation.
 3. Themethod of claim 1, wherein said craniofacial anatomical structuresinclude soft tissue of said patient.
 4. The method of claim 2, whereinsaid craniofacial structures comprise soft tissue, bone and dentition.5. The method of claim 2, wherein said first set of digital datacomprises a set of individual tooth models representing facialcomponents of the patient.
 6. The method of claim 1, further comprisingthe steps of (a) visually studying on said user interface theinteraction of said craniofacial anatomical structures and theirrelationship to the external, visual appearance of said patient; and (b)simulating changes in the anatomical position or shape of saidcraniofacial anatomical structures.
 7. The method of claim 6, whereinsaid step (b) of simulating changes in the anatomical position or shapeof craniofacial anatomical structures includes simulations of the effectof such changes on the external, visual appearance of said patient. 8.The method of claim 6, further comprising the steps of displaying thesmile of said patient; and viewing said smile after simulation ofchanging the position of said craniofacial anatomical structures.
 9. Themethod of claim 1, further comprising the steps of (a) moving saidcraniofacial structures; and (b) displaying the movement in a dynamicformat on a user interface.
 10. The method of claim 1, wherein saidfirst set of digital data is obtained from an in-vivo scan of facialcomponents and associated anatomical structures.
 11. The method of claim10, wherein said facial components comprise teeth.
 12. The method ofclaim 1, wherein said first set of digital data is obtained from a scanof a model of the patients' dentition, or a scan of a facial moulage.13. The method of claim 1, wherein said first set of data and saidsecond set of data are obtained from imaging devices selected from thegroup of imaging devices consisting of digital cameras, X-ray devices,hand-held 3-D scanners, laser scanners, computed tomography (CT)scanners, MRI scanners, coordinate measuring machines, destructivescanners, and ultrasound scanners.
 14. The method of claim 1, furthercomprising the step of storing a three-dimensional representation ofsaid patient's craniofacial structures in a format suitable for use by amanufacturer of orthodontic appliances.
 15. The method of claim 1,further comprising the step of transmitting said composite, combineddigital three-dimensional representation of said craniofacial anatomicalstructures over a computer network.
 16. The method of claim 1, whereinthe orthodontic treatment comprises a hybrid treatment in whichdifferent types of appliances (fixed or removable) are selected anddesigned to treat the patient.
 17. The method of claim 1, furthercomprising the step of simulating the changes in at least one of thefollowing: the age of the patient, the facial expression of the patient,and the coloring of the skin or teeth of the patient.
 18. The method ofclaim 1, further comprising the step of simulating the modification ofthe shape of any craniofacial structure.
 19. The method of claim 1,further comprising the steps of merging data representing athree-dimensional scan of the surface of the face of the patient anddata representing a two-dimensional color picture of the face of thepatient, thereby creating a three-dimensional colored virtual model ofthe face of the patient.
 20. The method of claim 1, wherein said firstand second sets of data are obtained from imaging devices selected fromthe group of imaging devices consisting of two dimensional cameras,X-ray devices, hand-held 3-D scanners, laser scanners, computedtomography (CT) scanners, MRI scanners, coordinate measuring machines,destructive scanners, and ultrasound scanners; and further comprisingthe steps of storing a third or more sets of data representing images,including one or more X-rays of said craniofacial structures, andsuperimposing, either automatically or with user involvement, thethree-dimensional model of said craniofacial structures from thecombination of said first and second data sets with said one or moreimages of said craniofacial structures.
 21. The method of claim 1,further comprising the step of integrating a patient management programhaving a scheduling feature for scheduling appointments for saidpatient.
 22. The method of claim 21, wherein said patient managementprogram provides a flexible scheduling of patient appointments based onprogress of treatment of said craniofacial anatomical structures. 23.The method of claim 22, wherein said progress of treatment is obtainedfrom periodically obtaining updated three-dimensional informationregarding the progress of treatment of the craniofacial features of thepatient.
 24. The method of claim 23, wherein said three-dimensionalinformation is obtained from scanning of the craniofacial anatomicalstructures with an optical, hand-held scanner.
 25. The method of claim23, wherein said craniofacial anatomical structures include the teeth ofthe patient.
 26. The method of claim 1, further comprising the step ofsuperimposing said first set of digital data and said second set ofdigital data with the use of an operator interaction, and wherein saidoperator interaction comprises: displaying said first and second sets ofdigital data on said user interface, said operator selecting points onsaid user interface in said first set of data which are common to saidsecond set of data.
 27. The method of claim 1, further comprising thestep of simulating changes in the position of the teeth of the patient.28. The method of claim 1, further comprising the step of simulatingchanges in soft tissue of the patient.
 29. The method of claim 1,wherein said composite, combined digital representation comprises atwo-dimensional representation.
 30. The method of claim 1, wherein saidcomposite, combined digital representation comprises a superposition ofa two-dimensional representation and a three dimensional representation.31. The method of claim 1, further comprising the step of modelinggrowth in craniofacial structure of functional movement of craniofacialstructures.